hold against
Think badly of someone because of some fault or occurrence. For example,
Even if you're late, I won't hold it against you
, or
She backed right into his new car, so he's bound to hold it against her
. Also see
bear a grudge